THE  UNIVERSITY 

0F  ILLINOIS 

LIBRARY 


370 
106 

no.i4H7 


-|wtrC3,tW!l 


A 


The  person  charging  this  material  is  re- 
sponsible for  its  return  to  the  library  from 
which  it  was  withdrawn  on  or  before  the 
Latest  Date  stamped  below. 

Theft,    mutilation,    and    underlining    of    books    are    reasons 
for    disciplinary    action    and    may    result    in    dismissal    from 
the  University. 
To  renew  call  Telephone  Center,  333-8400 

UNIVERSITY    OF    ILLINOIS    LIBRARY    AT    URBANA-CHAMPAIGN 


OCT  18 
SEP  27 


A 


1982 
1962 


L161— O-1096 


BULLETIN  NO.  14 


BUREAU  OF  EDUCATIONAL  RESEARCH 
COLLEGE  OF  EDUCATION 


THE  USE  OF  DIFFERENT 

TYPES  OF  THOUGHT  QUESTIONS  IN 

SECONDARY  SCHOOLS  AND  THEIR 

RELATIVE  DIFFICULTY 

FOR  STUDENTS 

by 

Walter  S.  Monroe, 
Director,  Bureau  of  Educational  Research 

and 

Ralph  E.  Carter 

Associate  Professor  of  Educational  Psychology 
Indiana  University,  Extension  Division 


PRICE   30  CENTS 


PUBLISHED  BY  THE  UNIVERSITY  OF  ILLINOIS,  URBANA 

1923  .  v 


— -Z^  £?  i 


/v-/y 


PREFACE 

This  bulletin  is  a  report  of  one  phase  of  a  larger  in- 
vestigation relating  to  the  study  habits  of  school  children. 
The  types  of  questions  asked  by  teachers  of  their  students 
are  important  both  because  of  the  mental  processes  which 
occur  in  answering  them  and  because  an  intimate  relation 
exists  between  the  questions  asked  and  the  detailed  objectives 
toward  which  the  students  work. 

Mr.  Carter  was  consulted  in  the  preparation  of  the 
questionnaire.  He  rendered  valuable  assistance  in  deciding 
upon  the  tentative  list  of  types  of  thought  questions.  Later, 
advantage  was  taken  of  his  presence  at  the  University  of  Illi- 
nois during  the  Summer  Session  of  1922  to  secure  assistance  in 
the  preparation  of  this  report.  He  is  largely  responsible  for 
its  general  organization. 

Walter  S.  Monroe,  Director. 

February  1,  1923 


539709 


Digitized  by  the  Internet  Archive 

in  2011  with  funding  from 

University  of  Illinois  Urbana-Champaign 


http://www.archive.org/details/useofdifferentty14monr 


The  Use  of  Different  Types  of  Thought  Questions 

in  Secondary  Schools  and  Their  Relative 

Difficulty  for  Students 

Recent  emphasis  upon  questioning  in  instruction.  The  first 
quantitative  study  of  the  questions  which  teachers  use  in  the  class- 
room was  made  by  Miss  Romiett  Stevens.1  In  this  analysis  of 
questions  a  distinction  was  made  between  those  that  required  memory 
only  and  those  that  required  thought.  Miss  Stevens'  discussion  has 
been  widely  quoted  and  has  been  very  influential  in  causing  teachers 
to  give  more  attention  to  the  types  of  questions  asked  of  their  students. 
It  is  altogether  likely  that  this  investigation  is  primarily  responsible 
for  a  material  increase  in  the  percent  of  thought  questions  which  are 
asked  in  our  schools  today.  Many  of  the  most  commonly  used  texts 
on  methods  of  teaching  devote  separate  space  to  the  topic  of  question- 
ing. Its  importance  has  been  especially  emphasized  by  Charters,2 
Strayer,3  Parker,4  and  Colvin.5 

Analysis  of  thought  questions.  An  examination  of  typical 
thought  questions  from  the  standpoint  of  the  mental  processes  which 
they  require  in  answering  reveals  certain  significant  differences.  Some 
questions  require  the  student  to  "compare"  two  or  more  ideas;  others 
ask  that  he  "summarize;"  still  others  demand  that  he  "give  reasons 
why."  The  probable  mental  processes  occurring  in  the  reflective 
thinking  called  for  rather  than  the  form  of  the  question  or  the  word- 
ing of  the  answer  have  been  made  the  basis  of  these  types  of  thought 


1Stevens,  Romiett.  "The  question  as  a  measure  of  efficiency  in  instruction," 
Teachers  College  Contributions  to  Education  No.  48.  New  York:  Teachers  College, 
Columbia  University,  1912. 

2Charters,  W  W.  Methods  of  Teaching.  Chicago:  Rowe  Peterson  and  Company, 
1912,  p.  444.   (Chapter  XVII). 

•■•Strayer,  G.  D.  A  Brief  Course  in  the  Teaching  Process.  New  York:  Macmillan 
Company,  1912,  p.  315.   (Chapter  XI) 

4Parker,  S.  C.  Methods  of  Teaching  in  High  Schools.  Boston:  Ginn  and  Company, 
1915,  p.  XXV,  529.   (Chapter  XX) 

6Colvin,  S.  S.  Introduction  to. High  School  Teaching.  New  York:  Macmillan 
Company,  1917,  p.  XXI,  451.    (Chapter  XV) 

[S] 


questions.  Altho  there  is  doubtless  considerable  overlapping  in  the 
types  of  mental  processes  which  usually  occur  in  answering  such 
questions,  there  appears  to  be  sufficient  differences  to  justify  the  re- 
cognition of  a  number  of  types  of  questions.  In  this  investigation 
the  following  tentative  list  of  types  of  thought  questions  was  used. 
A  more  refined  analysis  would  yield  probably  some  additional  types, 
but,  on  the  other  hand,  for  practical  purposes  it  is  possible  that  some 
combinations  of  types  might  profitably  be  made.  It  is  recognized 
that  the  mental  processes  of  different  students  doubtless  vary  in 
answering  the  same  question.  In  fact  a  thought  question  for  one  stu- 
dent may  be  a  memory  question  for  another  student,  or  even  for  the 
same  student  on  the  following  day. 

1.  Selective  recall— basis  given. 

Name  the  presidents  of  the  United  States  who  had  been  in  military  life  before 
their  election. 

What  do  New  Zealand  and  Australia  sell  in  Europe  that  may  interfere  with  our 
market? 

2.  Evaluating  recall — basis  given. 

Which  do  you  consider  the  three  most  important  American  inventions  in  the 
nineteenth  century  from  the  standpoint  of  expansion  and  growth  of  transportation? 

Name  the  three  statesmen  who  have  had  the  greatest  influence  on  economic 
legislation  in  the  United  States. 

3.  Comparison  of  two  things — on  a  single  designated  basis. 

Compare  Eliot  and  Thackeray  in  ability  in  character  delineation. 

Compare  the  armies  of  the  North  and  South  in  the  Civil  War  as  to  leadership. 

4.  Comparison  of  two  things — in  general. 

Compare  the  early  settlers  of  the  Massachusetts  Colony  with  those  of  the  Virginia 
Colony. 

Contrast  the  life  of  Silas  Marner  in  Raveloe  with  his  life  in  Lantern  Yard. 

5.  Decision — for  or  against. 

Whom  do  you  admire  more,  Washington  or  Lincoln? 

In  which  in  your  opinion  can  you  do  better,  oral  or  written  examinations? 

6.  Causes  or  effects. 

Why  has  the  Senate  become  a  much  more  powerful  body  than  the  House  of 
Representatives? 

What  caused  Silas  Marner  to  change  from  what  he  was  in  Lantern  Yard  to  what 
he  was  in  Raveloe? 

7.  Explanation  of  the  use  or  exact  meaning  of  some  phrase  or  state- 
ment in  a  passage. 

8.  Summary  of  some  unit  of  the  text  or  of  some  article  read. 

[6] 


9.  Analysis.    (The  word  itself  is  seldom  involved  in  the  question.) 

What  characteristics  of  Silas  Marner  make  you  understand  wrwfRavflgejDeople 
were  suspicious  of  him?  '  "  R   i 

Mention  several  qualities  of  leadership. 

10.  Statement  of  relationships. 

Why  is  a  knowledge  of  Botany  helpful  in  studying  agriculture? 
Tell  the  relation  of  exercise  to  good  health. 

11.  Illustrations  or  examples  (your  own)  of  principles  in  science,  con- 
struction in  language,  etc. 

"12.  Classification.    (UsuallytheconverseofNo.il). 

What  is  the  principle  involved  here?  What  is  the  construction?  To  what  class 
or  genus  does  this  individual  belong? 

13.  Application  of  rules  or  principles  in  new  situations. 

14.  Discussion. 

Discuss  the  Monroe  Doctrine. 
Discuss  early  American  Literature. 

15.  Statement  of  aim — author's  purpose  in  his  selection  or  organiza- 
tion of  material. 

What  was  the  purpose  of  introducing  this  incident? 
Why  did  he  discuss  this  before  that? 

16.  Criticism — as  to  the  adequacy,  correctness,  or  relevancy  of  a 
printed  statement,  or  a  classmate's  answer  to  a  question  on  the 
lesson. 

17.  Outline. 

18.  Reorganization  of  facts.  (A  good  type  of  review  question  to  give 
training  in  organization.) 

The  student  is  asked  for  reports  where  facts  from  different  organizations  are 
arranged  on  an  entirely  new  basis. 

19.  Formulation  of  new  questions — Problems  and  questions  raised. 
What  question  came  to  your  mind? 

What  else  must  be  known  in  order  to  understand  the  matter  under  consideration? 

20.  New  methods  of  procedure. 

Suggest  a  plan  for  proving  the  truth  or  falsity  of  some  hypothesis. 

How  would  you  change  the  plot  in  order  to  produce  a  certain  different  effect? 

Relation  to  educational  objectives  of  types  of  questions  asked 
by  teachers.  Incidentally  it  may  be  noted  that  the  types  of  questions 
used  both  for  stimulating  and  directing  the  mental  activity  of  the 
learner  and  for  measuring  the  results  of  teaching  reflect  in  a  subtle 
way  the  educational  objectives  of  a  teacher.  The  objectives  thus 
indicated  may  not  agree  with  those  stated  by  the  teacher  but  they 
nevertheless  are  an  index  of  the  objectives  toward  which  the  students 

[7] 


direct  their  efforts.  In  studying,  students  tend  to  prepare  to  answer 
the  kind  of  questions  which  they  think  the  teacher  will  ask.  It  is 
altogether  likely  that  the  study  objectives  of  students  are  influenced 
more  by  the  kind  of  questions  asked  than  by  direct  statements  of 
aims  in  the  course. 

Purpose  and  method  of  this  investigation.  The  purpose  of  this 
study,  which  is  Sub-project  II  of  our  investigation  of  the  study 
habits  of  high  school  pupils,  was  announced  as  follows:  "To  deter- 
mine the  extent  of  the  use  of  different  types  of  thought  questions  in 
actual  school  practise  and  the  relative  difficulty  of  these  types  for 
students."  The  method  employed  was  to  submit  a  questionnaire  to  a 
large  number  of  teachers  in  the  seventh  and  eighth  grades  and  in  the 
high  school.  In  this  questionnaire  the  list  of  the  twenty  types  of 
thought  questions  given  on  pages  6-7  was  reproduced.  The  teachers 
were  urged  to  study  these  types  carefully  before  answering  any  of  the 
questions.  It  was  pointed  out  that  an  attempt  to  classify  a  question 
according  to  this  plan  might  lead  to  the  conclusion  that  it  is  a  combina- 
tion of  two  or  three  types.  However,  in  such  a  case,  it  is  probably 
true  that  part  of  the  work  of  answering  has  been  done  by  the  author 
in  the  textbook,  leaving  to  the  student  only  that  phase  of  the  question 
which  would  definitely  classify  it  as  belonging  to  one  of  the  above 
types. 

Plan  of  summarizing  data.  Usable  replies  were  received  from  199 
teachers  representing  almost  every  school  subject  and  all  parts  of  the 
state.  A  few  of  these  replies  did  not  give  answers  to  one  or  more  of 
the  questions  but  they  were  included  in  the  tabulation  for  the  other 
questions.  By  subject  the  number  of  replies  were  as  follows:  English, 
41;  History,  48;  Science,  41;  Mathematics,  31;  Foreign  Language,  12; 
Geography,  11;  Commercial  subjects,  6;  Agriculture,  4.  Taking  only 
the  replies  from  teachers  of  English,  History,  and  Science  there  were 
26  for  grades  VII  and  VIII  combined;  34  for  grades  IX  and  X;  and 
45  for  grades  XI  and  XII.  It  was  thought  advisable  to  summarize 
separately  the  replies  for  English,  History  and  Science.  Those  re- 
ceived from  teachers  of  all  other  subjects  were  grouped  together. 
The  replies  to  Questions  I  and  IV  obviously  do  not  lend  themselves 
to  statistical  treatment.  The  data  yielded  by  the  other  three  ques- 
tions are  summarized  in  the  following  tables. 

Limitations  of  replies  to  the  questionnaire.  Several  teachers 
took  occasion  to  mention  difficulties  which  they  experienced  in  filling 
out  the  questionnaire  blank.   That  some  difficulties  were  encountered 

[8] 


is  not  surprising  because  the  concept  of  different  types  of  thought 
questions  was  undoubtedly  new  to  most,  if  not  all,  of  the  teachers. 
The  limitations  of  space  prevented  an  extensive  description  of  each 
type.  Undoubtedly  teachers  differed  in  their  interpretations  of  the 
various  types.  Furthermore,  in  answering  Question  II  they  were 
asked  not  only  to  differentiate  between  thought  questions  and  mem- 
ory questions  but  also  to  classify  the  thought  questions  under  the 
various  types.  In  doing  this,  they  had  to  rely  upon  their  memory  of 
the  questions  they  had  asked.  For  these  reasons  the  replies  to  Ques- 
tion II  must  be  thought  of  as  giving  only  a  very  rough  indication  of 
the  practise  of  teachers  with  reference  to  the  types  of  questions  which 
they  asked.  A  few  teachers  reported  that  their  replies  were  based  on  a 
careful  analysis  of  lists  of  questions  which  have  been  used  and  pre- 
served. If  these  lists  were  reasonably  complete  their  replies  should  be 
more  accurate  than  those  received  from  teachers  who  relied  upon  their 
memory  of  the  questions  they  had  asked. 

It  may  be  pointed  out  that  other  methods  of  ascertaining  the 
relative  frequency  of  use  of  different  types  of  thought  questions  would 
not  eliminate  all  difficulties.  For  example,  if  a  trained  investigator 
should  visit  the  classes  of  199  teachers  in  various  parts  of  the  state  a 
sufficient  number  of  times  to  obtain  a  fair  sample  of  the  habitual 
practise  and  should  try  to  make  a  .record  of  all  types  of  questions 
asked  he  would  encounter  difficulties  in  definitely  classifying  them. 
It  is  not  possible  to  judge  accurately  of  the  mental  processes  stimu- 
lated by  a  question  unless  one  is  acquainted  with  the  previous  ex- 
periences of  the  pupils  in  the  field  of  this  question.  This  has  been 
referred  to  already  in  pointing  out  that  what  constitutes  a  thought 
question  for  one  pupil  may  be  merely  a  memory  question  for  another. 
The  analysis  of  stenographic  reports  of  lessons  would  present  certain 
difficulties  for  the  same  reason.  The  teacher  knows  better  than  any 
one  else  what  acquaintance  a  student  has  with  a  topic  and  what 
the  textbook  states  about  it.  For  this  reason  he  has  the  advantage  of 
a  casual  observer. 

Additional  types  of  questions  mentioned.  Question  I  was  asked 
in  order  to  obtain  suggestions  for  supplementing  the  list  of  type 
questions  for  further  study  in  this  field  and  also  to  give  each  teacher 
an  opportunity  to  report  fully  his  own  practise  in  the  use  of  different 
types  of  thought  questions  in  case  he  did  not  consider  the  list  ade- 
quate. The  replies  to  the  questionnaire  show  that  the  number  of 
teachers  who  did  not  report  the  use  of  all  of  the  types  given  in  the  list 

[9] 


UNIVERSITY  OF  ILLINOIS 
BUREAU  OF  EDUCATIONAL  RESEARCH 

COLLEGE  OF  EDUCATION 

Types  of  Thought  Questions 

Name Address 

School  Subject School  grade 

I.  Write  in  on  the  blanks  at  the  bottom  of  the  list  below  any  additional  types  of 
thought  questions  you  use  to  any  great  extent  in  the  grade  and  subject  you  have 
chosen.     (Include  these  additional  types  in  answering  all  other  questions.) 

II.  What  percent  of  all  of  your  questions  to  the  class  are  of  each  type  listed?  (See 
directions,  p.  11).     Put  the  percent  of  each  type  in  the  column  on  the  right. 


Types  of  Thought  Questions 


Percent  of  all 
questions 


1. 

2. 
3. 
4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
12. 
13. 
14. 
15. 
16. 
17. 
18. 
19. 
20. 
21. 
22. 
23. 
24. 


Selective  recall — basis  given 

Evaluating  recall — basis  given 

Comparison  of  two  things — on  a  single  designated  basis. 

Comparison  of  two  things — in  general 

Decision — for  or  against 

Causes  or  effects 

Explanation 

Summary 

Analysis 

Statement  of  relationships , 

Illustration  or  examples 

Classification 

Application 

Discussion 

Statement  of  aim 

Criticism 

Outline 

Reorganization  of  facts 

Formulation  of  new  questions 

New  methods  of  procedure 


III.  In  the  left  hand  margin  make  an  "X"  opposite  each  of  the  five  types  for  which 
students  answers  are  least  likely  to  be  satisfactory.  (See  directions,  p.  1 1) 

IV.  Choose  three  of  the  types  you  marked  with  an  "X."   State  for  each  some  of  the 
students'  most  common  faults  in  procedure  in  answering  it.    Mention  faults  of 


[10] 


omisson  as  well  as  faults  of  commission.    Mention  only  faults  peculiar  to  this 
type  of  question.  Use  other  side  of  this  sheet  if  necessary.   (See  directions  below.) 

V.     If  we  consider  all  unsatisfactory  answers  made  by  students  in  school  work,  what 
percent  of  them,  in  your  opinion,  are  due  primarily  to  bad  habits  of  procedure 

in  answering  questions?  Give  separate  replies  for  memory  questions % 

and  thought  questions % 

The  following  detailed  directions  for  answering  the  questionnaire 
were  given: 

Before  answering  any  of  the  questions,  decide  what  subject  and  what  grade 
(any  from  VII  to  XII)  you  will  have  in  mind  in  all  of  your  replies.  Answer  the  questions 
with  reference  to  only  one  school  subject.  Fill  in  the  blanks  at  the  top  of  the  first  page 
of  the  questionnaire. 

Question  I.  Examine  the  list  of  types  given  above  to  see  if  there  are  not  some 
other  types  of  thought  questions  that  you  use  to  a  considerable  extent.  Altho  the  given 
list  may  seem  long  and  inclusive  at  first,  it  is  not  complete. 

Question  II.  Before  you  put  down  your  estimates  for  Question  II,  you  should 
decide  what  is  your  proportion  of  thought  questions  and  of  pure  memory  questions. 
This  question  calls  for  a  further  analysis  of  your  thought  questions.  It  may  be  advis- 
able to  make  rough  estimates  for  all  before  you  fill  in  the  data  on  the  questionnaire. 
After  you  have  the  differences  between  the  types  in  mind,  make  the  best  estimate  you 
can,  even  tho  you  may  not  be  very  sure  in  many  cases.  You  are  merely  declaring  your 
best  judgment,  not  guaranteeing  that  it  is  infallible.  We  want  your  opinion  of  what 
your  practise  is,  not  oj  what  you  think  it  ought  to  be. 

Question  HI.  In  Question  III,  we  are  concerned  with  the  process  of  answering 
so  we  must  assume  that  the  student  has  the  information  necessary  for  a  satisfactory 
answer  if  he  will  only  use  it  as  the  type  of  the  question  requires.  The  common  starting 
point  for  an  informal  diagnosis  of  a  student's  study  faults  is  in  his  unsatisfactory 
answers.  An  unsatisfactory  answer  may  indicate  lack  of  information  or  certain  faulty 
habits  in  answering  questions.  There  are  many  cases  in  which  teachers  find  out  by 
other  means  that  the  student  has  enough  information  to  give  a  satisfactory  answer 
but  has  failed  because  he  did  not  take  the  necessary  steps  and  precautions  in  replying 
to  the  question.  When  some  students  correct  examination  papers  returned  to  them, 
they  are  heard  to  say,  "I  knew  that,  but  I  didn't  think  of  it,"  or  "I  knew  that  but  I 
didn't  know  that  was  what  the  question  meant."  Some  of  the  faults  in  procedure  are 
common  to  all  kinds  of  questions;  others  are  peculiar  to  particular  types.  In  Question 
III,  you  are  asked  to  decide  on  the  difficulty  of  the  types  for  students.  In  other  words, 
in  which  type  do  they  have  the  worst  habits  of  procedure? 

Question  IV.  It  will  be  helpful  in  answering  Question  IV  to  think  of  some  par- 
ticular questions  of  the  type  under  consideration  and  then  express  the  students'  faults 
in  fairly  general  terms. 

Question  V.  As  long  as  the  unsatisfactory  answer  is  a  resultant  of  poor  methods 
of  preparation  and  poor  habits  of  answering  questions,  we  must  correct  the  latter  or 
take  it  into  consideration  in  inferring  what  methods  of  study  are  needed.  It  is  important 
to  get  an  estimate  from  a  large  number  of  teachers  of  the  percent  of  failures  (in  answers) 
that  are  due  primarily  to  bad  habits  of  procedure  in  answering  questions. 

[11] 


is  very  much  greater  than  the  number  mentioning  additional  types 
Only  thirty-nine  teachers  (19.6  percent)  indicated  that  they  used  all 
twenty  types  of  questions.  Thirteen  teachers  supplemented  the  list, 
each  writing  in  from  one  to  three  additional  types.  These  teachers 
were  distributed  among  the  different  subjects  as  follows:  seven  in 
History,  three  in  English,  two  in  Science,  and  one  in  Mathematics. 
Only  four  of  those  suggesting  additional  types  of  questions  indicated 
that  they  used  all  twenty  of  the  types  given  in  the  questionnaire  list. 

Some  of  the  additional  types  of  questions  suggested  are  clearly 
included  in  the  list  given  in  the  questionnaire.  The  following  are 
typical:  (A)  "Connections  between  historical  events,"  Type  4  or  6; 
(B)  "Cause  and  results,"  included  in  Type  6;  (C)  "Application  to  the 
present  time,"  included  in  Type  13;  (D)  "Determining  the  significant 
word  in  a  statement,"  included  in  Type  19.  Other  suggested  types  are 
not  so  clearly  included  but  several  of  them  under  a  fairly  loose  inter- 
pretation may  be  placed  with  the  types  listed.  Illustrations  of  these 
are:  (A)  "Imagining  the  results  if  conditions  were  different,"  a 
special  case  of  Type  6;  (B)  "Questions  on  appreciation  such  as,  which 
do  you  like  best?"  a  fairly  common  form  of  Type  2;  (C)  "Identify  a  ' 
known  character  in  a  scene  in  the  story  when  his  name  is  not  mention- 
ed by  the  author,"  a  special  case  of  Type  12;  (D)  "What  should  you 
judge  from  these  facts?"  This  last  is  broad  enough  to  cover  many 
types.  Frequently,  it  would  fall  under  Type  6.  There  were  a  few 
suggestive  questions  which  are  not  so  easily  classified  under  the  20 
types  given.  Some  of  these  are:  (A)  "Trace  the  development," 
mentioned  by  two  teachers;  (B)  "Estimate  the  importance  of";  . 
and  (C)  "Why  is  this  statement  true?"  a  very  common  question  in 
geometry.  By  a  very  liberal  interpretation  these  might  be  put  under 
Types  9,  2,  and  6  respectively  but  it  is  likely  that  many  teachers 
would  consider  them  sufficiently  different  to  justify  naming  them  as 
additional  types. 

The  answers  to  the  first  question  indicate  that  the  list  of  types 
is  reasonably  complete.  For  practical  purposes  it  is  probably  better 
to  give  a  rather  loose  interpretation  in  classifying  special  cases  than 
to  try  to  extend  the  list.  Largely  for  this  reason  the  authors  decided 
to  limit  the  summary  of  the  replies  to  the  other  parts  of  the  ques- 
tionnaire to  the  original  twenty  types. 

Frequency  of  use  of  different  types  of  questions.  Question  II  of 
the  questionnaire  was  asked  in  order  to  secure  answers  for  the 
following: 

[12] 


*^o 


1.  Which  types  of  questions  are  most  commonly  used  by  teachers 
in  classroom  instruction? 

2.  Which  types  of  questions  are  most  characteristic  of  the  in- 
struction in  different  subjects? 

3.  Do  teachers  in  the  more  advanced  grades  tend  to  ask  different 
types  of  questions  from  those  in  the  lower  grades? 

Table  I  gives  a  detailed  summary  of  the  replies  by  forty-eight 
teachers  of  History.  This  table  shows  that  four  of  these  teachers  did 
not  report  any  use  of  questions  requiring  selective  recall  (Type  1); 
one  teacher  reported  that  this  type  of  question  formed  1  percent  of 
all  his  questions;  five  teachers  indicated  that  this  type  made  up  2 
percent  of  all  the  questions  which  they  asked;  eleven  named  5  percent 
as  an  index  of  its  frequency;  eight  teachers  considered  that  15  per- 
cent of  their  questions  were  of  this  type;  and  another  eight  teachers 
indicated  that  more  than  15  percent  of  their  questions  called  for 
selective  recall.   The  median  practise  is  7.5  percent. 

The  outstanding  characteristic  of  the  table  is  the  variability  of 
practise  which  it  indicates.  In  the  case  of  each  type  of  question  there 
were  one  or  more  teachers  who  gave  no  indication  of  its  use.  For 
several  of  the  types  the  number  of  teachers  not  indicating  any  use  is 
surprisingly  large.  Furthermore,  there  is  no  type  of  question  which 
did  not  receive  an  indication  of  at  least  5  percent  by  one  teacher. 
Most  of  the  types  received  an  indication  of  10  percent  or  more.  Thus, 
if  we  may  assume  that  the  replies  to  this  portion  of  the  questionnaire 
are  a  reasonably  true  indication  of  practise,  it  is  clear  that  teachers 
of  History  vary  widely  in  the  types  of  thought  questions  which  they 
ask  of  their  pupils  and  hence  necessarily  vary  widely  in  the  detailed 
objectives  which  their  pupils  strive  to  attain. 

Similar  tabulations  were  made  for  English,  Science  and  other 
subjects  combined.  The  median  frequencies  for  the  four  groups  of 
subjects  are  given  in  Table  II.  If  we  consider  only  English,  History 
and  Science,  students  are  most  commonly  required  to  answer  the 
following  types  of  thought  questions:  cause  and  effect,  Type  6; 
selective  recall,  Type  1;  discussion,  Type  14;  and  evaluating  recall, 
Type  2.  The  types  of  questions  which  are  asked  least  frequently  are: 
formulation  of  new  questions,  Type  19;  new  methods  of  procedure, 
Type  20;  and  reorganization  of  facts,  Type  18. 

Different  subjects  require  different  mental  processes.  An  ex- 
amination of  Table  II  reveals  that  the  frequency  of  use  of  some  of  the 

[13] 


OS 

DC 

o 

cr> 

OS 
to 

U 

< 

w 

H 

oo 

m 

Q 

w 

en 
< 

O 

M 

H 
en 
W 

8 

H 
DC 
(J 
D 
O 
DC 
H 

to 
O 

C/3 

W 

CL, 
>H 

H 


e 

r3 

tncOCOw->sO''fvOlJ">l'">©©OV©co©'OfNCOOO© 

T3 

r-r^-iou->eN©eN»ococO>-i           h    o    «  M   «i   M           — 

s 

I* 

«^n 

OO    CO                               O             CO                               —             ON                      — 

O 

w» 

00*0-— i<N>— i    tF   cN    — i                  i-i                  VO    <-i                  i-h 

"""' 

■* 

~-                                                                                    ri 

*"* 

CO 

O] 

"■" ' 

CN 

co    —                  —                                -i 

C 

o 

CO 

3 

^H 

c 

4-t 

O 

\0"ivi^MPinONNH           CNCN           Ot    ?i    *    «    h 

X 

60 

3 

O 

J= 

H 

00 

«  cs  *  -h  n        ^^^^Tf         —         co        ~h        oi^^h 

>-*- 

o 

c 

r^- 

CNCO                   -    N                   ^h-^                   ^hc^^h 

<u 

Oh 

VO 

CS             «S             <N    i-h                      Tf-HCN«^ 

>j-> 

— •    r~'^Oco©K>OvOvy">    — '•>+'\0v>0eNN0C4O^-m           CO 

r-^                  ^^      ^—      i     1                             .     1      *—                                                                                         i      1 

■* 

CO    ■*    CN    CO    CO    ■— iCN'-o,^i-*,CNCO-HTtiTj<Tf           coco 

eo 

(N    •*    —    CO'*1'— i    OO    h    t    "t    N   M    ri           co-*'t'\ot^CN 

cn 

vi'*TfMiflrt(»Noobn\ois'- 1  vo  n  "i  m  n  os 

^h^hui^<-h          rf<   ,— i   t-»   *f   m   'f   wi          h   >o   to   oo   t^   ui 

rH 

o 

*N,o*0'^lou^^to,'l»0'*'t,<^|c,^^o^,'-^' 

H             ^                      rt    M    Ol    N             CN    — '             m    «    (N 

to 

C 

o     • 

CO 

to 

to 

.2 

a, 

2    = 

CO 

U 

£ 

«s  *   : 

aj 

o 

CO 

0 

recall 
l — sin 
i— gei 

CO 
o 

£    r 

C 

U 

°C8 
O 

tion  o 
n  of  n 
ds... 

U 

tive  re 
uating 
parisoi 
parisoi 

«    o 

°    ««    - 
to     C     « 

y   m   £ 

•J 
> 

tionshi 
tration 
sificati 
ication 
ussion. 
;ment 

ganiza 
lulatio 
methc 

coWUUQuW7)<ctisu<Q(flUOei<to2 

F<(Nn'i,'i'i\oiNOoo\d«rl(i'*l'isoh'oo'o\0 

[14] 


TABLE  II.     MEDIAN  FREQUENCY  OF  USE  OF  DIFFERENT  TYPES  OF 

THOUGHT  QUESTIONS  IN  ENGLISH,  HISTORY,  SCIENCE  AND 

OTHER  SUBJECTS 


Types  of  Questions 


Median  percent  of  Use 


English 


History 


Science 


Others 


1.  Selective  recall 

2.  Evaluating  recall 

3.  Comparison — single 

4.  Comparison — general 

5.  Decision — for  or  against 

6.  Causes  or  effects 

7.  Explanation 

8.  Summary 

9.  Analysis 

10.  Relationships 

11.  Illustration  or  examples 

12.  Classification 

13.  Application 

14.  Discussion 

15.  Statement  of  aim 

16.  Criticism 

17.  Outline 

18.  Reorganization  of  facts 

19.  Formulation  of  new  questions 

20.  New  methods 


5.4 
5.9 
5.4 
5.0 
4.4 
5.8 
10.0 
5.6 
5.7 
1.8 
3.6 
1.6 
2.9 
5.7 


7.5 

7.3 

5.3 

5.5 

2.6 

10.4 

2.6 

5.5 

3.5 

3.0 

.96 

.9 

0 

3 

0 

6 

2 

3 


2.1 


1.0 


5.3 
2.8 
3.9 
3.1 
1.1 
10.2 
3.3 
2.9 
1.8 
5.7 
8.6 
5.2 
5.1 
5.3 
.7 
2.9 


1.7 
1.5 


5.1 

.96 
1.6 
1.9 
1.2 
4.8 
5.1 
2.1 
3.1 
3.4 
5.1 
5.1 
5.5 
1.1 

.8 
3.9 

.8 

.9 
1.8 
3.6 


types  of  thought  questions  varies  in  the  different  subjects.  For 
example,  Type  7  "Explanation  of  the  use  or  exact  meaning  of  some 
phrase  or  statement  in  a  passage"  is  the  most  frequently  used  type 
of  question  in  English  but  it  is  eleventh  in  frequency  of  use  in  His- 
tory. Asking  students  to  give  an  illustration  or  example,  Type  11, 
is  next  to  the  most  frequent  type  of  question  in  Science  but  it  is  sel- 
dom used  in  History.  Type  10,  which  asks  the  student  to  state  a  re- 
lationship, is  frequently  used  in  Science  but  is  infrequently  asked  of 
students  in  English.  Summaries  and  outlines  are  required  more 
frequently  in  History  and  English  than  in  the  other  subjects.  Thus, 
if  our  assumption  that  different  types  of  questions  require  different 
mental  processes  is  valid,  we  have  here  evidence  that  different  types 
of  mental  processes  are  required  of  students  in  the  different  subjects. 
Hence,  we  should  expect  to  find  that  the  problem  of  directing  the 
learning  of  students  is  different  in  different  subjects. 


[IS] 


TABLE  III.    PERCENT  OF  TEACHERS  REPORTING  NO  USE  OF  CERTAtt 
TYPES  OF  THOUGHT  QUESTIONS 


Types  of  Questions 


Grades 
VII-VIII 


Grades 
IX-X 


Grades 
XI-XII 


1.  Selective  recall 

2.  Evaluating  recall 

3.  Comparison — single 

4.  Comparison — general 

5.  Decision — for  or  against 

6.  Causes  or  effects 

7.  Explanation 

8.  Summary 

9.  Analysis 

10.  Relationships 

11.  Illustration  or  examples 

12.  Classification 

13.  Application 

14.  Discussion 

1 5.  Statement  of  aim 

16.  Criticism 

17.  Outline 

18.  Reorganization  of  facts 

19.  Formulation  of  new  questions 

20.  New  methods 


3 

6 
18 

9 
21 

9 
23 

6 
21 
35 
41 
50 
46 

9 
63 
26 
26 
35 
68 
44 


18 

29 
9 
18 
32 
6 
26 
23 
23 
18 
26 
29 
35 
15 
46 
23 
21 
41 
44 
53 


11 
13 
18 
13 
27 
4 
18 
22 
29 
20 
16 
24 
22 
13 
42 
24 
31 
42 
44 
32 


Variations  in  the  use  of  type  questions  in  different  school  grades. 

The  replies  of  113  teachers  have  been  summarized  in  Table  III  on  the 
basis  of  the  school  grades  in  which  they  were  teaching.  The  subjects 
involved  are  English,  History,  Science  and  Geography.  Instead  of 
giving  the  median  frequency  of  use,  the  percent  of  teachers  indicating 
no  use  of  the  given  type  of  question  has  been  calculated.  Thus,  3 
percent  of  the  teachers  in  the  seventh  and  eighth  grades  reported  no 
use  of  selective  recall.  In  the  ninth  and  tenth  grades  18  percent  of  the 
teachers  did  not  use  this  type  and  in  the  eleventh  and  twelfth  grades, 
11  percent.  The  general  impression  prevails  that  students  in  ad- 
vanced classes  are  asked  more  difficult  types  of  questions  than  stu- 
dents in  the  lower  grades.  A  comparison  of  the  data  for  the  three 
grade  groups  shows  that  the  differences  are  not  very  great.  In  gen- 
eral, it  appears  that  the  variations  in  the  use  of  thought  questions  are 
greater  for  different  subjects  than  for  the  different  grades. 

The  relative  difficulty  of  the  different  types  of  questions  for 
students.    In  Question  III  teachers  were  asked  to  designate  the  five 


[16] 


TABLE  IV.  TYPES  OF  QUESTIONS  FOR  WHICH  STUDENTS'  ANSWERS 
ARE  LEAST  SATISFACTORY 


Types  of  Questions 

English 

History 

Science 

Others 

Total 

Rank 

1.  Selective  recall 

2.  Evaluating  recall 

3.  Comparison — single...  . 

4.  Comparison — general .  . 

5.  Decision — for  or  against 

6.  Causes  or  effects 

7.  Explanation 

5 

12 

6 

12 

1 

17 

15 

10 

13 

5 

5 

5 

16 
10 
15 
8 
6 
12 

9 

6 

10 

12 
7 

12 
J9 
17 
12 
14 
15 
17 
4 
4 
12 
14 
12 
14. 
12 
19 

6 

7 

8 

11 

9 

9 

2 

19 

15 

9 

11 

10 

10 

9 

18 

5 

5 

7 

6 

13 

8 

7 

14 
10 

5 
14 

6 
24 
25 
14 
21 
20 
14 
16 
29 
10 
10 
10 

9 
19 

18 
24 

37 
45 
27 
47 
18 
77 
67 
47 
60 
42 
33 
34 
77 
39 
42 
39 
33 
63 

41 
44 

15 
8 

19. 
6 

20 
1 
3 

8.  Summary 

6 

9.  Analysis 

5 

10.  Relationships 

10 

11.  Illustration  or  examples 

17 
16 

13.  Application 

1^ 
13 

14.  Discussion 

15.  Statement  of  aim 

16.  Criticism 

10 
13 

17.  Outline 

17 

18.  Reorganization  of  facts. 

19.  Formulation  of  new 

questions 

4 
12 

20.  New  methods 

9 

types  of  questions  in  which  "students'  answers  were  least  likely  to  be 
satisfactory."  A  summary  of  their  replies  is  given  in  Table  IV.  The 
first  line  of  this  table  should  read  as  follows:  Questions  calling  for 
selective  recall  were  named  among  the  five  most  difficult  types  by  five 
teachers  of  English,  ten  teachers  of  History,  eight  teachers  of  Science, 
and  fourteen  teachers  of  other  subjects.  This  type  of  question  ranks 
fifteenth  in  the  frequency  of  mention  among  the  five  most  difficult 
types.  The  two  types  most  frequently  mentioned  were  Type  6, 
causes  and  effects,  and  Type  13,  application.  Both  of  these  were 
mentioned  by  seventy-seven  teachers  or  38.7  percent.  Altho  we  may 
say  that  in  the  judgments  of  those  answering  the  questionnaire  these 
two  types  are  the  most  difficult  for  students,  it  should  be  noted  that 
over  60  percent  of  the  teachers  did  not  list  them  among  the  five  most 
difficult  types. 

Here,  as  in  other  tables,  the  variation  in  the  replies  of  teachers 
is  very  extreme.  Even  when  we  consider  the  teachers  of  a  single 
subject  we  find  marked  variations  in   their  judgments  concerning 


[17] 


the  difficulty  of  the  different  types  of  questions.  No  type  of  question 
is  so  easy  that  it  is  not  included  among  the  most  difficult  five  by 
several  teachers.  This  variation  in  judgment  is  probably  due  largely 
to  the  fact  that  teachers  have  not  recognized  the  distinction  in  types 
of  questions  which  are  made  here  and,  furthermore,  they  have  not 
analyzed  the  responses  of  their  students  in  order  to  ascertain  the 
nature  of  the  difficulties  which  the  students  encounter  in  answering 
the  questions. 

Common  faults  of  procedure  in  answering  different  types  of 
questions.  In  Question  IV  the  teachers  were  asked  to  select  three  of 
the  types  which  they  considered  most  difficult  and  to  state  for  each 
of  these  some  of  the  students'  most  common  faults  in  procedure  in 
answering  them.  An  examination  of  the  replies  to  this  question 
suggests  that  a  considerable  number  of  teachers  are  satisfied  with 
telling  students  that  the  answer  is  wrong,  or  what  the  answer  should 
have  been,  and  that  they  fail  to  give  much  thought  to  the  students'  ' 
faults  of  procedure  in  answering  the  question.  It  may  be  pointed  out 
that  questions  are  only  a  means  to  an  end  and  this  end  is  the  education  <| 
of  the  student.  The  answer  is  in  itself  relatively  unimport- 
ant. The  question  fulfils  its  function  only  when  it  stimulates 
educative  processes  in  the  mind  of  the  student.  If  the  mental  processes 
which  the  question  initiates  are  not  educative  the  asking  of  the  ques- 
tion has  been  largely  futile.  Hence,  it  becomes  highly  important  for 
the  teacher  to  give  attention  to  the  procedure  which  the  student  em- 
ploys in  answering  questions  in  order  that  the  faults  of  procedure  may 
be  corrected  so  that  the  student's  mental  processes  will  become  most 
effective  in  his  education. 

Of  the  199  teachers  from  whom  replies  to  the  questionnaire  were  re- 
ceived, twenty-three  did  not  attempt  to  answer  Question  IV  at  all  and 
twenty-four  others  discussed  all  three  types  of  questions  together  or 
stopped  after  discussing  only  one  or  two.  Furthermore,  a  number  of 
other  teachers  gave  answers  which  show  that  they  failed  to  grasp  the 
significance  of  this  question.  In  substance  they  said  that  the  difficulty 
in  students'  procedure  in  answering  a  question  is  inability  to  do  what 
is  called  for.  For  example,  one  said  that  the  fault  in  answering  Type  2, 
evaluating  recall,  is  that  "pupils  are  not  always  able  to  evaluate." 
Another  said  that  in  questions  on  aim  "they  seem  unable  to  see  any 
motive  behind  the  statement  of  the  author."  Still  another  said  that 
the  fault  in  Type  6,  causes  and  effects,  is  that  "they  are  unable  to  see 
causes  and  effects."    Still  others  mentioned  objective  faults  in  the 

[18] 


answers  rather  than  faults  in  procedure.  For  example,  one  teacher 
mentioned  that  "minor  details  are  included  while  leading  points 
are  omitted."  Another  stated  that  "applications  are  not  as  good  as 
could  be  expected."  Still  another  teacher  simply  stated  that  "the 
answers  are  inaccurate."  General  faults,  such  as  "lack  of  practise  in 
answering  questions,"  "carelessness,"  "lack  of  concentration,"  "pupils 
do  not  know  how  to  study,"  and  so  on  were  mentioned  by  one  or  more 
teachers. 

It  may  be  pointed  out  that  teachers  who  fail  to  become  definitely 
conscious  of  the  difficulties  which  their  students  encounter  will  be 
unable  to  give  them  much  definite  constructive  assistance  in  this 
phase  of  their  learning.  Probably  the  most  significant  conclusion  to 
be  drawn  from  the  answers  to  this  part  of  the  questionnaire  is  that 
many  teachers  are  failing  to  give  attention  to  the  procedure  which 
students  use  in  answering  questions.  They  appear  to  be  concerned 
largely  with  the  accuracy  of  the  answer  and  when  it  is  wrong  they 
fail  to  seek  the  cause  in  the  procedure  which  the  student  has  used. 

Several  of  the  faults  which  appear  to  be  suggestive  are  given 
below.  This  list  is  not  a  complete  statement  of  the  faults  of  students 
in  answering  questions  but  should  prove  helpful  to  teachers  who  are 
desirous  of  ascertaining  the  reasons  why  students  fail  to  answer 
questions  satisfactorily. 
Type  2. 

The  pupil  fails  to  see  real  basis  on  which  evaluation  should  be  made. 

Type  3. 

A  student  seldom  stays  "on  a  single  designated  basis." 
Type  3  or  4. 

The  pupil  begins  to  answer  the  question  before  he  has  thought  out  what  points 
should  be  included  in  the  comparison.  He  often  includes  minor  comparisons  and  omits 
the  important  points. 

Unless  there  is  a  single  designated  basis  for  comparison,  pupils  fail  to  see  all  of  the 
possibilities  of  comparison.  The  slothful  pupil  is  content  with  one  or  two  obvious  com- 
parisons. 

The  pupil  can  state  some  information  about  each  thing  compared,  but  omits 
the  comparative  connection  between  the  two  things.  The  pupil  gives  comments  on 
one  thing  and  omits  the  discussion  on  the  other  thing,  showing  how  the  two  are  alike 
comparatively. 

In  making  comparisons,  students  usually  give  characteristics  of  things  to  be 
compared,  or  describe  them  separately  but  fail  to  make  a  complete  comparison. 
Type  4. 

In  comparing  two  things  in  general  the  worst  fault  is  in  the  students'  forgetting 
some  of  the  important  points  to  be  compared.  Often  enough  suggestions  must  be 
given  so  that  the  question  finally  should  be  placed  under  Type  3. 

[19] 


Pupils  frequently  fail  to  distinguish  the  important  from  the  trival.  The  pupil' 
prejudices — likes  and  dislikes — influence  the  comparison. 

Type  6. 

Students  confuse  cause  and  effect.  Method  requires  real  thinking  and reasoning — 
the  hardest  thing  any  student  can  be  asked  to  do. 

Students  often  know  a  statement  is  true  but  on  account  of  lack  of  thoroness  can 
not  give  causes  or  reasons. 

Type  6  or  7. 

Pupils,  when  asked,  "Why?"  seem  to  forget  all  reasoning  and  usually  state  the 
first  thing  that  comes  to  mind  and  fail  also  to  go  far  enough  in  explanation. 

Type  7. 

Pupils  have  difficulty  in  finding  words  to  express  the  thought  without  repeating 
word  for  word  parts  of  the  passage  to  be  explained.  The  chief  difficulty  seems  to  be  the 
limitations  of  their  vocabulary. 

The  trouble  here  is  that  pupils  seem  to  have  failed  to  study  the  statement  under 
question  sufficiently  and  so  have  quite  an  indefinite  notion  of  the  words  used.  And  too, 
even  if  they  succeed  in  getting  a  fairly  good  understanding  of  the  text  read,  they  are 
too  careless  of  the  choice  of  their  own  words  in  reproducing  the  meaning  exactly. 
They  are  too  hasty  in  the  study  of  the  passage  and  have  not  yet  been  trained  to  go 
slowly  and  get  facts  one  at  a  time. 

Type  8. 

A  discussion,  like  a  summary  often  lacks  point.  The  student  fails  to  organize 
material  and  grasp  essentials. 

Students  fail  to  properly  organize  data  mentally  before  expressing  them. 

Type  9. 

Here  pupils  are  often  prone  to  feel  satisfied  if  only  a  few  of  the  factors  or  qualities 
involved  in  the  analysis  are  learned  and  reproduced.  Many  times  but  one  factor  will 
be  offered  when  to  completely  analyze  the  problem  will  require  several.  Being  allowed 
to  "get  off"  with  a  minimum  of  thinking  is  one  of  the  chief  causes  of  poor  analysis. 

Type  10. 

Pupils  often  fail  to  get  the  facts  correlated  before  they  attempt  to  state  the  exist- 
ing relationships. 

Pupils  do  not  stop  to  analyze  in  order  to  see  the  relationship  but  answer  without 
giving  much  thought. 

Pupils  answer  various  separate  facts  instead  of  showing  relationship.  These  facts 
are  usually  true  and  have  a  bearing  upon  the  question  but  the  actual  relationship  is 
usually  missing  in  the  answer.  That  is,  the  pupil  does  not  seem  to  see  how  a  change  in 
one  fact  would  influence  some  other  related  fact. 

Type  11. 

Illustrations  or  examples.  This  is  not  so  difficult  after  the  student  has  had  some 
practise,  but  at  first  he  always  tries  to  find  his  illustrations  in  some  field  remote  from 
his  own.  It  takes  him  a  while  to  realize  that  scores  of  illustrations  come  daily  under  his 
observation — ;/  he  observes. 

[20] 


Type  12. 

In  classifying,  the  pupils  make  their  answers  too  readily  without  first  trying  one 
class  then  another  or  without  thinking  whether  the  one  they  choose  actually  does  fit 
better  and  why. 

The  student's  failure  in  Type  12  is  due  mainly  to  his  failure  in  Type  7.  He  fails 
to  decide  correctly  upon  a  construction  because  he  has  not  thought  out  carefully  the 
exact  meaning  of  the  sentence.   This  in  turn  is  due  to  mental  laziness  or  carelessness. 

Type  14. 

He  discusses  each  phase  at  randcm  and  separately  without  relating  it  to  others. 
A  discussion   like  a  summary  often  lacks  point.    The  student  fails  to  organize 
material  and  grasp  essentials. 

Students  fail  to  properly  organize  data  mentally  before  expressing  them. 
Students  do  not  group  their  facts  together  sufficiently  to  discuss  them  properly. 

Type  17. 

A  pupil  doesn't  really  outline  into  main  facts  with  heads  under  them,  but  makes 
long  statements  one  after  another,  or  puts  steps  in  before  their  time. 

General. 

Pupils  will  not  read  and  get  a  thoro  knowledge  of  the  subject-matter  first. 

With  Types  8,  14,  and  17  pupils  do  not  summarize  or  discuss  or  outline  in  a  way 
to  make  clear  to  another  the  things  under  discussion.  They  are  not  logical  in  the  ar- 
rangement of  their  ideas.  They  answer,  at  least  85  percent  of  them,  only  well  enough 
so  that  one  who  knows  already  can  tell  that  they  know,  but  only  about  15  percent 
can  arrange  and  subordinate  well  enough  to  be  clear  to  the  unintiated. 

A  common  fault  is  careless  reading  of  the  question  (or  hearing  it  as  the  case 
may  be).  This  is  especially  true  in  Algebra.  Another  fault  is  "saying  the  first  thing  that 
comes  into  mind"  instead  of  carefully  considering  its  application  to  the  problem  under 
discussion.  In  Geometry,  for  example,  the  student  does  not  rapidly  recall  all  theorems 
pertaining  to  the  subject  in  hand,  eliminate  those  inapplicable,  and  so  discover  the 
correct  one. 

Percent  of  unsatisfactory  answers  due  to  faulty  procedure  in 
answering  questions.  In  Question  V  the  teachers  were  requested  to 
indicate  separately  for  memory  questions  and  for  thought  questions 
the  percent  of  unsatisfactory  answers  which,  in  their  opinion,  were 
due  primarily  to  bad  habits  of  procedure.  A  summary  of  their  re- 
plies is  given  in  Table  V.  As  previously  indicated  it  appears  certain 
that  a  considerable  number  of  teachers  have  given  relatively  little 
thought  to  the  faults  in  the  procedure  used  by  students  in  answering 
questions.  Consequently,  the  replies  to  this  question  must  be  con- 
sidered as  representing  no  more  than  rough  estimates.  In  the  case 
of  thought  questions  the  median  percent  of  unsatisfactory  answers 
due  to  faulty  procedure  is  approximately  50  for  all  of  the  subjects. 
In  the  case  of  memory  questions  faulty  procedure  is  considered  to  be 
a  less  potent  cause.     There  are  two  factors  that  probably  tend  to 

[21] 


C/5 

z 

o 

H 
in 

W 

P 

c 

o 
z 

5 
w 

Z 

< 

»— ( 

W 

OS 
D 
Q 
W 
U 

OS      § 

—  "B 

D    e 

<  l"H 

fa        o 

o  -* 

r*     « 

D  -5 
Q    $ 

%  •* 

—  V 

fa      a 

£  £ 

Z     o 

<  U 

^    < 

OS     w 

o 

H 
U 

:- 
< 

fa 

o 

I-H 

z 
w 
u 

os 
w 

Oh 


w 

PQ 

< 
:- 


■- 


-H  ^-  r<  —  Tf     fSJ 


—  (N    « 


p^_v^3^H\coJr^--* 


io         —   ~ 


-  m  «  oo  n 


~  W) 


t  "i  c  t  n 


«       fV)       ^       (Vj       -^« 


vo  r~  — <  o)  — 


W 


cso,oor--\0'*<r^ooo 


— i  {N  -* 


•^ocr^r-.^or^co— to 


n  "i  n  "i 


C-l    — '   <N    c*5 


W 


—  —        n  i^ 


r«-  <->  <n  r~-  cj 


[22] 


make  these  estimates  lower  than  they  should  be.  Many  teachers 
showed  by  their  replies  to  Question  IV  that  they  had  very  little  ex- 
perience in  analyzing  answers  for  the  purpose  of  determining  the  cause 
of  faulty  answers.  Naturally  they  could  not  be  expected  to  recognize 
the  real  extent  of  such  faults  of  procedure  on  the  part  of  the  members 
of  their  classes.  Other  teachers  use  such  a  limited  range  in  types  of 
questions  that  they  would  not  have  a  chance  to  observe  as  many 
faults  in  answer  technique  as  teachers  who  use  a  greater  variety  of 
types  of  questions. 

Relation  between  frequency  of  use  and  difficulty.  There  seems 
to  be  very  little  relation  between  the  difficulty  of  a  type  and  the 
frequency  of  its  use.  Type  6,  causes  and  effects,  is  the  most  commonly 
used  type  and  at  the  same  time  it,  together  with  Type  13,  application, 
is  considered  the  most  difficult  for  students.  Only  one  of  the  five 
types  reported  as  most  difficult,  Type  18,  reorganization  of  facts, 
is  among  the  five  least  frequently  used;  only  one  of  the  five  types 
reported  as  least  difficult,  Type^S^comparison  on  a  single  basis,  is 
among  the  five  most  frequently  used.      S£-»_^x-*  (^  ^jf1*  ^ 

Relation  of  frequency  of  use  of  types  of  questions  to  emphasis 
upon  different  types  of  learning.  An  earlier  bulletin1  of  the  Bureau 
of  Educational  Research  reports  the  relative  emphasis  on  different 
types  of  textbook  study-  in  the  various  subjects.  We  now  have  some 
data  on  the  question,  "How  does  the  frequency  of  use  of  types  of 
questions  correspond  with  the  emphasis  on  the  different  types  of  text- 
book study  required  of  students?"  Some  of  the  types  of  study 
recognized  in  the  report  are  too  broad  to  be  tested  by  any  one  type 
of  thought  question.  There  are,  however,  question  types  that  cor- 
respond rather  closely  with  certain  of  the  study  types. 

To  get  a  rough  measure  of  the  relation  of  the  frequency  of  use  of 
the  different  types  of  thought  questions  to  the  emphasis  upon  the 
types  of  learning,  we  may  arrange  the  types  in  order  of  frequency  of 
use  and  divide  the  list  into  quartiles  (four  equal  parts).  For  instance, 
in  English,  Study  Type  II  is  in  the  first  quartile  (i.  e.  the  three  most 
frequently  required  study  types);    Study  Type  X  is  in  the  fourth 


Monroe,  Walter  S.  "Types  of  learning  required  of  pupils  in  the  seventh  and 
eighth  grades  and  in  the  high  school."  University  of  Illinois  Bulletin,  Vol.  19,  No.  15, 
Bureau  of  Educational  Research  Bulletin,  No.  7,  Urbana:  University  of  Illinois,  1921, 
16.  p. 

[23] 


Types  of  Textbook  Study 


Types  of  Questions 


II.  Summary  of  central  ideas  in  the 
lesson  studied. 

III.  Prepare  an  outline  of  principal  points 
and  supporting  details  in  the  lesson 
arranged  to  show  order  of  relative 
importance  and  relationship. 

VI.  Discovery  of  collateral  or  illustrative 
material  for  topics  or  problem  under 
discussion. 

VIII.  Appreciation  of  the  significance  of 
each  word  used  in  a  concisely  ex- 
pressed statement  or  principle. 

IX.  A  clear  comprehension  of  the  essen- 
tial conditions  of  a  problem  which  is 
to  be  solved. 

X.  Discovery  of  new  or  supplementary 
problems  related  to  the  topic  being 
studied. 


8.  Summary, 


17.  Outline. 


11.  Illustration  or  examples. 


7.  Explanation  of  the  use  or  meaning  c 
some  phrase  or  statement  in  a  passagt 

9.  Analysis. 


19.  Formulation  of  new  questions. 


quartile  (i.  e.  one  of  the  least  frequently  required  study  types).7 
Table  VI  shows  the  extent  of  agreement  in  rank  in  frequency  of  use 
of  some  corresponding  types  of  study  and  questions.  The  first  column 
gives  by  number  the  corresponding  types  as  listed  above.  The  first' 
number  given  in  the  columns  for  the  separate  subjects  designates  the 
quartile  rank  of  the  study  type  on  the  same  horizontal  line;  the, 
second  number  designates  the  rank  of  the  corresponding  type  of 
question.  It  will  be  noted  that  the  rank  agrees  in  seven  out  of  eighteen 
possible  cases.  In  English  the  disagreement  is  greatest  in  the  case  of 
questions  calling  for  analysis  (first  rank)  and  study  requirements  of  a 
clear  comprehension  of  the  conditions  of  a  problem  (fourth  rank); 
in  History,  the  discovery  of  collateral  or  illustrative  material  ranks 
second  in  study  requirements  but  questions  calling  for  illustrations  or 
examples  rank  fourth  in  use;  in  Science,  the  same  study  requirement 
ranks  fourth  and  the  corresponding  question  type  ranks  first.  Thirty- 
three  percent  of  the  Physics  and  Chemistry  teachers  in  the  first  ques- 
tionnaire mentioned  Study  Type  X,  discovery  of  new  or  supple- 
mentary problems,  as  a  major  type  of  study  in  these  subjects,  yet 
44  percent  of  the  teachers  of  these  subjects  by  their  admission  do  not 
ask  any  questions  of  Type  19,  formulation  of  new  questions,  to  their 


'These  facts  are  derived  from  Table  II  of  the  Bulletin  referred  to  above. 

[24] 


CO 

o 

H 

CD 

C 

O 

co 
W 
O, 
>« 
H 

Q 
< 

Q 
D 
H 

co 

O 

co 
W 

CL, 
tH 

H 
fa 
O 

u 

Q 
Z 

2 

co 

£* 
a! 
O 
U 


>" 

l 

3 

.-            to 

Q 

co        I 

£ 

+    + 

Jm 

.3         2? 

d 

< 

I        W 

!> 

j 

«         ._;         >-3 

>< 

X          CO          oo 

< 

^+      4-      + 

> 

60     Ml              60             ti 

c    c          c         .2 

UJ  UJ        UJ        I 

w 

'0 

u 

co 

O 
< 

+ 

*J        .      60    £        .60 

*^  .j* 

u 

CO    ■*    •— i    CN    CO    CO 

o 

•z. 

w 

CJ 

co 

.£•-* 

-a    c 

CO    •*    •*    CO    CO    CN 

*3  _k: 

1  s 

CN    CS    Tf    CO    CN    ■*■ 

t- 

C/X 

a: 

o 

M 

ffi 

£••* 

3     rt 

CN    -h    cN    co    if    co 

co* 

«   ^i 

|  s 

CN    CO    CO    — i    r->    Tf 

s 

a* 

c/3 

O 

2: 

w 

-£■  ■* 

^3     C 
3      P3 

—    CN    CO    CN    ■*■    «*• 

CO    — 

o 
z 

•Si 

1) 

co    f^    — i    t^-    Cs    OS 

3 

C 

o    a 

^    - 

^_ 

a; 

o 

co 

—    —    •> 

[25] 


high  school  juniors  and  seniors.  It  would  seem  from  the  data  that 
some  teachers  may  not  be  using  the  types  of  questions  that  would 
stimulate  and  test  students'  achievements  in  some  of  the  types  of 
study  they  are  expected  to  use. 

Conclusions.  Probably  the  most  significant  conclusions  to  be 
drawn  from  this  investigation  are:  (1)  teachers  are  not  sufficiently 
conscious  of  the  types  of  questions  which  they  are  accustomed  to  ask 
and  of  the  significance  of  these  types,  and  (2)  in  general  teachers  do 
not  analyze  unsatisfactory  answers  to  questions  in  order  to  ascertain 
whether  such  answers  are  due  to  a  faulty  technique  on  the  part  of 
the  student. 

A  number  of  other  conclusions  are  worthy  of  mention.  (1) 
Teachers  of  the  same  subject  vary  widely  in  the  extent  of  their  use  of 
different  types  of  thought  questions.  (2)  The  frequency  of  the  use  of 
a  given  type  does  not  seem  to  depend  very  much  on  (a)  the  school 
grade,  (b)  the  subject,  or  (c)  the  supposed  difficulty  of  the  type.  It  is 
very  likely  that  some  teachers,  who  expect  their  students  to  use  cer- 
tain types  of  study,  do  not  use  the  types  of  questions  that  are  best 
suited  to  test  their  students'  achievements  due  to  those  particular 
types  of  study.  (3)  IfTeachers  individually  think  that  certain  types  of 
questions  are  more  difficult  for  students  than  others.  However,  there 
is  no  very  great  agreement  among  teachers  as  to  the  relative  diffi- 
culty of  the  various  types.jj 


[26] 


BULLETIN  NO.  15 


BUREAU  OF  EDUCATIONAL  RESEARCH 
COLLEGE  OF  EDUCATION 


THE    CONSTANT   AND  VARIABLE   ERRORS 
OF  EDUCATIONAL  MEASUREMENTS 


by 

Walter  S.  Monroe 
Director,  Bureau  of  Educational  Research 


PRICE  25  CENTS 


PUBLISHED  BY  THE  UNIVERSITY  OF  ILLINOIS,  URBANA 

1923 


